Adaptive loss-feedback loop + reactive ARQ - #3
Merged
Conversation
Closes the data-plane control loop: an authenticated receiver->sender Control packet carrying delivered-count + missing object counters drives both (1) the class-aware controller (bulk/default repair ratio earns zero on clean flows, activating the dormant FEC bypass + halving datagrams; realtime keeps a floor) and (2) deadline/class-aware reactive ARQ (sender retransmits fresh RaptorQ repair symbols for eligible NACKed objects). Receiver reports raw gaps; sender owns all class attribution and retransmit eligibility. Plan sequences feedback first (throughput checkpoint), then ARQ. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8 tasks in two phases. Phase A (control channel + feedback + class-aware zero-repair): LossReport (de)serialization, gap-based loss detector, controller that lets ARQ classes earn ratio 0, daemon control-channel wiring, and the Phase-A throughput verdict. Phase B (reactive ARQ): bounded retransmit buffer, retransmit-on-NACK with fresh repair symbols reusing the original object_id, plus the final measurement. Counter is one unified per-direction sequence; sender owns class attribution; ARQ + zero-repair gated on FlowParams.arq. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implement the feedback module for the receiver-to-sender loss report. Includes serialization to big-endian wire format with 14-byte header (delivered_count:u32, high_counter:u64, n_missing:u16) followed by missing sequence numbers (u64 each). Decode validates input length and rejects malformed bytes. Encode caps missing at MAX_NACK (64). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements LossDetector in crates/yip-transport/src/lossdetect.rs: BTreeMap<u64,u64> pending set (counter → first-implied-ms), grace window before declaring missing, window-bounded eviction of smallest keys. Four tests green; full yip-transport suite 35/35; clippy clean. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both peers now send a sealed Control packet every 30 ms containing a LossReport (delivered_count, high_counter, missing counters). The receiver decrypts the report, looks up each missing counter in a bounded sent-log (counter → FlowClass, capacity 4096), computes per-class loss fractions, and feeds them to Transport::observe_loss, closing the feedback loop into the adaptive FEC controller. Changes: - handshake.rs: add PacketType::Control = 3 - tunnel.rs: SentLog ring (capacity 4096), LossDetector Arc<Mutex>, egress records counter→class in sent_log, ingress calls on_seen for every datagram (data and control) and on_delivered when FEC decodes, emits feedback every FEEDBACK_INTERVAL_MS=30, handles incoming Control packets to call observe_loss per class; periodic log of the bulk controller repair ratio every 5 s. - yip-transport/lib.rs: add Transport::bulk_repair_ratio() for diagnostics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Feedback loop converges the bulk controller to repair ratio 0.0000 (confirmed via yipd diagnostic log), firing the FEC-encode bypass and halving per-packet datagrams. Clean-link single-stream TCP rises from ~273-285 to ~457 Mbit/s (~60%), consistent with removing the ~24us encode (80% of egress CPU) + 2->1 datagrams. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduces `RetxBuffer` in `crates/yip-transport/src/retxbuf.rs`: a `HashMap<u64, Entry>` + `VecDeque<u64>` order list that holds at most `max` ciphertext objects keyed by send-counter, evicting the oldest entry (by insertion order) on cap overflow and skipping expired entries (age > ttl_ms) on `get`. Mirrors the `FlowTable` pattern in `flow.rs`. The stored `object_id: u16` is preserved so ARQ retransmits reuse the receiver's existing FEC decoder for that object rather than starting a new one. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- FecEncoder::repair_with_id: encode with an explicit object_id so the receiver's existing FEC decoder is topped up (not a new decode object). - Transport::repair_object: thin wrapper used by the daemon. - Unit test: retransmitted_repair_completes_a_missing_object (stall then top-up with extra_repair=4 symbols carrying the original object_id). - tunnel.rs egress: after encode(), put (counter, ciphertext, class, oid) into the shared RetxBuffer (TTL 2s, cap 1024). - tunnel.rs ingress control handler: after observe_loss, iterate report.missing; for each Bulk-class counter in the retx buffer, call repair_object and send the repair datagrams. Lock discipline enforced: retx lock is never held across transport or io calls. - run-netns-tunnel-loss.sh + ping_across_yipd_tunnel_under_loss: netem 10% loss on both veth legs; 10 pings must all succeed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ve-repair rationale The Default class emits a proactive repair symbol, so a 2-source object encodes to 3 symbols; skipping one still decodes. Deliver exactly one symbol to guarantee the object is genuinely incomplete before the retransmit tops it up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Defense-in-depth hardening: LossReport::decode now caps the decoded missing vec at MAX_NACK entries regardless of peer-supplied count, bounding retransmit work per report. Added decode_caps_missing_at_max_nack test that constructs raw bytes for a 100-entry report and verifies successful decode with truncation to MAX_NACK (64). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase A throughput win holds (bulk ratio 0.0000, clean-link ~457 Mbit/s). Loss recovery is unchanged with the feedback loop live: yip delivers 99.7%/99.0% UDP at 5%/10% netem loss (controller re-arms FEC on loss). Reactive ARQ retransmits Bulk NACKs with fresh repair symbols reusing the original object id; codec is unit-proven, wiring reviewed, tunnel survives 10% loss. A dedicated end-to-end Bulk-ARQ integrity harness is a tracked follow-up. Recorded in the bench README and CHANGELOG. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… yip-transport coverage to >=90% Fix 1 (security): in tunnel.rs ingress, the control-packet path called detector.on_seen(counter, now_ms) BEFORE session.open(counter, ct), letting a forged packet poison the loss detector's high_counter even when authentication fails. Move on_seen to run only inside the Ok branch of session.open. Data-path on_seen calls are unchanged. Fix 2 (coverage): yip-transport line coverage was 89.53%; add four focused unit tests in fec.rs covering repair_with_id, the late/duplicate symbol guard (state.done), the SBN out-of-range guard for already-tracked objects, and the full-Encoder encode path. Line coverage: 90.05%. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 1, 2026
This was referenced Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the data-plane control loop and activates the throughput win the throughput pass (PR #2) set up. Spec + plan committed; executed as 8 reviewed tasks (subagent-driven).
What landed
Controlpacket (PacketType::Control=3, sealed via the session AEAD) carrying{delivered_count, high_counter, missing[]}. Newyip-transportmodules:feedback(LossReport),lossdetect(gap-basedLossDetector),retxbuf(RetxBuffer).Bulk) flows decay their repair ratio to zero on a clean link (firing the dormant FEC-encode bypass + halving datagrams);Realtime/Defaultkeep a proactive floor. Loss snaps the ratio back up instantly.Bulkobjects with fresh RaptorQ repair symbols reusing the originalobject_id(tops up the receiver's existing decoder);Realtime/Default/expired objects are not retransmitted.Results (release, netns, kernel 6.18 / Ryzen 5 7640U)
0.0000(yipd diagnostic log); clean-link single-stream TCP ~273–285 → ~457 Mbit/s (~60%).Review
Per-task reviews + a final whole-branch opus review. The final review found and we fixed one security must-fix (control-path
on_seennow runs only after AEAD authentication, so a forgedControlpacket can't poison the loss detector) and raisedyip-transportcoverage to 90.05%. Cross-task counter semantics, ARQ object-identity (u16 wrap unreachable within the bounded window), whole-loop lock discipline (no deadlock, no lock across I/O), auth-before-trust, and bounded state all verified sound.Follow-ups (documented, non-blocking): a dedicated end-to-end Bulk-ARQ integrity harness (establish tunnel before applying loss); the gap detector only reports fully-absent counters (exact for the single-symbol zero-repair case).
🤖 Generated with Claude Code